Collective communication in which a plurality of processors participate can be divided into scatter collective communication, in which one processor serving as a root scatters data to all other processors, and gather collective communication, in which one processor serving as the root gathers data from all processors. Data is sent and received between the root processor and other processors in both of types of collective communication. If this is realized in simple fashion, data need only be sent individually between the root processor and other processors.
When such processing is actually carried out, however, the processors execute communication serially, the result of which is greatly degraded performance. In general, therefore, collective communication is performed through parallel processing as much as possible by using a binary-tree approach. More specifically, as shown in FIG. 7, which illustrates schematically the scattering of data in scatter collective communication, it is known that the expansion and gathering of data in a plurality of processors proceed in steps of the square of 2, whereby high-speed processing can be achieved. The system comprises 16 processors, among which those indicated by the black circles are processors that possess data and those indicated by the white circles are processor that wait for data. At step 1, the first data possessing processor, which is located at the extreme left, delivers data to the neighboring second processor by communicating with this processor. At step 2, the first and second processors possessing data each skip one processor to communicate with third and fourth processors, respectively, that are waiting for data. As a result, the first through fourth processors come to possess data. Thus, at step 4, the eight processors on the left side communicate with the eight processors on the right side. As a result, all 16 processors come to possess data in four steps.